Automatic Parsing and Recognition of Hand-drawn Sketches for Pen-based Computer Interfaces

Pen-based computer interaction is becoming increasingly ubiquitous as evidenced by the growing interest in Tablet PC’s, electronic whiteboards and PDA’s. Many of these devices now come equipped with robust hand-writing recognizers. However, a problem that remains largely unsolved is the recognition of graphical input such as schematic sketches and diagrams. When faced with such input, these devices either leave the pen strokes uninterpreted, or offer only limited support for recognition, while placing many unnatural constraints on the way the user draws. These constraints might include limitations to single-stroke objects, or the need for user involvement in separating different visual objects. In this work, we present a new approach for recognizing hand-drawn, diagrammatic sketches. The key advance is an integrated sketch parsing and recognition model designed to enable natural and fluid pen-based computer interaction. With this approach, the stream of pen strokes is first examined to identify certain delimiter patterns called “markers.” These then anchor a spatial analysis which groups the remaining strokes into distinct clusters, each representing a single visual object. Finally, a shape recognizer is used to find the best interpretations of the clusters. This approach eliminates many of the unnatural constraints imposed by existing sketch understanding systems. To demonstrate our techniques, we have built SimuSketch, a sketch-based interface for Matlab’s Simulink package, and VibroSketch, a sketch-based interface for analyzing vibratory mechanical systems. In both systems, users can construct functional engineering models by simply sketching them on a computer screen. Users can then interactively manipulate their sketches to change model parameters and run simulations. Our user studies have indicated that even novice users can effectively utilize these systems to solve real engineering problems, without having to know much about the underlying recognition techniques.